The conf_zfs=openzfs build linked the legacy BSD-ZFS kstat stub
(bsd/.../opensolaris_kstat.o) to satisfy kstat_create/install/delete,
but every OpenZFS module was compiled against the OSv SPL kstat_t in
external/openzfs/include/os/osv/spl/sys/kstat.h (~64 bytes: ks_data,
ks_ndata, ks_data_size, ks_flags, ks_update, ks_private, ks_private1,
ks_lock).  The BSD stub allocated only a 16-byte kstat_t, so OpenZFS
callers such as arc_init()/dnode_init() wrote ks_update/ks_private past
the end of the allocation, corrupting the malloc free list.  The next
kstat_create() then returned a garbage pointer and faulted at
ks_ndata = ndata (SIGSEGV -> abort with empty backtrace right after the
version banner, before mkfs ran).

Fix: implement OSv-native kstat_create/install/delete in
openzfs_osv_compat.c using the correct OpenZFS kstat_t layout (virtual
kstats; OSv has no /proc or sysctl consumer), and filter the BSD
opensolaris_kstat.o out of the openzfs solaris object list in the
Makefile so only the correct implementation is linked.

The zfs_builder guest now boots, runs mkfs (creates + mounts pool osv),
populates it via cpiod, and exports cleanly.

…mount

zfs_unmount() (called by zfs destroy, zpool export, and property
remounts) only unmounts a dataset if it can find it via
libzfs_mnttab_find() -> libzfs_mnttab_update() -> getmntent().  On OSv
that lookup went nowhere for two reasons:
 - MNTTAB pointed at a nonexistent path, so fopen(MNTTAB) failed before
   getmntany() ever ran; and
 - the getmntent/getmntany stubs in libzfs_util_os.c returned EOF.
Datasets auto-mounted by the kernel (zfs_domount at pool/dataset create
time, not through the libzfs do_mount path) were therefore invisible to
libzfs, the objset stayed owned, and zfs destroy / zpool export failed
with EBUSY / 'dataset is busy'.

Fix, as two edits to the OpenZFS patch series:
 - patch 0004: MNTTAB -> /etc/mnttab (created empty at boot, so fopen
   succeeds and libzfs proceeds to getmntany()).
 - patch 0020: add lib/libzfs/os/osv/libzfs_mnttab_os.cc implementing
   getmntent/getmntany over the live VFS mount table via
   osv::current_mounts() (filtered to ZFS), replacing the EOF stubs.

Wire the new C++ shim into the libzfs build, filtering the C-only
warning flags out of ozfs-cflags-common that a C++ translation unit
rejects.  Makes zfs destroy and zpool export/import work on OSv, which
the feature-coverage tests depend on.

zpool export/import intermittently aborted with

  Assertion failed: owner.load(...) == sched::thread::current()
  (core/lfmutex.cc: unlock: 221)

in condvar::wait().  Root cause: the OpenZFS userland thread pool
(lib/libtpool) hands jobs to worker pthreads that block in
pthread_cond_wait() on a shared tp_mutex.  On OSv a pthread mutex and
condvar ARE the kernel lockfree::mutex + condvar, whose wait-morphing
protocol transfers mutex ownership from the signalling thread to a
waiter.  During tpool_destroy() teardown that handoff races: a worker
returns from pthread_cond_wait() and re-enters it, unlocking a tp_mutex
it no longer owns -> lfmutex owner assertion.  Reproduced deterministically
on zpool import, whose device-scan (zutil_import.c) and mount
(libzfs_mount.c) thread pools default to hundreds of workers
(mount_tp_nthr = 512).

Fix: run tpool jobs synchronously on OSv (0021) and force serial dataset
mounting (0022).  OSv threads are cheap and these jobs are short, so
serial execution removes the worker/teardown machinery and the race at no
practical cost.  Verified 0/8 asserts (was 8/8) across create/export/
import/status cycles from a clean patch-series rebuild.

…dataset)

Patch 0017 reclaimed only the mount-root znode before dmu_objset_disown.
A mounted child dataset (e.g. pool/fs) leaves its own znodes live, each
holding an object bonus buffer that pins a dnode.  dnode_destroy never
runs, the objset os_dnodes list never empties, and spa_export() blocks
forever in spa_evicting_os_wait().  Reproduced deterministically by:
zpool create; zfs create pool/fs; zpool export (hang).

Walk z_all_znodes at unmount and inactivate every remaining znode, as
vop_inactive would, so the objset drains.  Verified 6/6 completions
(was 0/6, hung) for create+child-fs+export+import+destroy.

OSv read_partition_table() names MBR slots 0-based (first slot is
/dev/vblkN.0) and exposes a raw unpartitioned disk directly as /dev/vblkN
with no child node.  zfs_append_partition() unconditionally appended .1, so
\x27zpool create test /dev/vblk1\x27 on a raw wiped NVMe failed with
\x27cannot open /dev/vblk1.1: No such file or directory\x27.

Fix (patch 0023): only append .1 when that partition node actually exists;
otherwise use the whole raw disk as-is, matching Linux/FreeBSD whole-disk
behavior (ZFS writes its own labels to the whole device).

Also adds FINDINGS-osv-openzfs.md documenting Bug 1, Bug 2, and the
page-allocator-under-memory-pressure investigation (>=4G runtime guests).

Verified: zpool create on a raw disk now succeeds, pool ONLINE.

…r 0)

zpool list faulted with strlen(NULL) in print_line() because
zpool_prop_column_name() returned NULL for a default list column on OSv.
The pool data row is correct; only the header label resolves NULL.
Guard header against NULL, falling back to the ZFS missing-value glyph,
so the command cannot crash.

zfs_is_readonly() was hardcoded B_FALSE, so a readonly=on dataset stayed
writable on OSv.  Read the readonly property at mount into a new
zfsvfs->z_readonly, return it from zfs_is_readonly(), and reject the write
vnops (write/create/remove/rename/mkdir/rmdir/setattr/truncate/symlink)
with EROFS.  Verified writes fail EROFS after remount and readonly=off
restores writability.

vdev_disk_open() never set vd->vdev_has_trim, so zpool trim/autotrim
reported trim not supported.  Enable it optimistically (OSv has no
capability query): the ZIO_TYPE_TRIM->BIO_DISCARD path reclaims space on
virtio-blk with discard, and devices without discard get ENOTSUP mapped
gracefully.  Verified zpool trim completes 100% with discard=unmap.

…ryption/dedup/draid/TRIM/O_DIRECT/etc)

Head-to-head storage microbenchmark of the two ZFS ports on identical
a bare-metal host, 8 GiB KVM guest, single-vdev (raw NVMe) and raidz2
(7x20 GiB). No Postgres/HammerDB - isolates the filesystem layer.

Key finding (the ARC-bridge measure): mmap of a 512 MiB file costs BSD-ZFS
~5 MiB extra RAM vs OpenZFS ~501 MiB - BSD's unified ARC/page-cache bridge
shares pages, OpenZFS double-buffers via its borrow path. Same root cause
drives BSD's warm-read and cached-random-read wins. OpenZFS wins the
write/CPU paths (seq write +34%, lz4 +56%, metadata +63%, fsync +11%) and
uniquely offers O_DIRECT (ARC bypass, ~raw-ceiling, BSD has no direct=).

Harness (scripts/bench/): pure-C zfs-bench.c (a g++ .so pulls libstdc++
symbols OSv can't resolve at load; C avoids GLIBCXX/CXXABI/__isoc23). Runs
off the zfs_builder bootfs - no usr.img cpiod populate. rebuild-bench.sh,
run-bench.sh, run-all.sh drive it; results.tsv is the raw data.

Copyright (C) 2026 Greg Burd

…en_zfs/

Per review: external/ is reserved for things compiled into the kernel
(libfdt, acpica).  The vendored OpenZFS tree is a module dependency, so
move the submodule from external/openzfs to modules/open_zfs/openzfs
alongside its patch series and module.py.

Updates: .gitmodules path (still points at the published upstream
github.com/openzfs/zfs at tag zfs-2.4.2, no fork), the OZFS/OPENZFS make
variables (Makefile + bsd/sys/cddl/openzfs_sources.mk), the parse-time
patch-apply step (stamp path, -d guard, and the git -C ... apply path,
whose relative patch prefix changes from ../../modules/open_zfs/patches
to ../patches now that the submodule sits one level deeper), and the
doc comments that named the old path.

… zfs

Restructure the in-kernel ZFS build into the module layout requested in
PR cloudius-systems#1423 review (issue cloudius-systems#1201), mirroring how java is provided by
openjdk*-from-host:

  - modules/zfs        placeholder; module.py selects the impl by conf_zfs
                       (openzfs -> open_zfs, else -> bsd_zfs). Still ships the
                       shared libsolaris.so via usr.manifest.
  - modules/bsd_zfs    provides = ["zfs"]. Owns the legacy BSD/Illumos ZFS
                       build rules in bsd_zfs_sources.mk (the solaris compat
                       list, the zfs core list, and the conf_zfs=bsd
                       object/flag selection) moved out of the top-level
                       Makefile.
  - modules/open_zfs   provides = ["zfs"]. openzfs_sources.mk moved here as
                       open_zfs_sources.mk and extended with the
                       conf_zfs=openzfs object/flag selection (solaris +=
                       $(openzfs-all), kstat filter, OpenZFS CFLAGS).

The top-level Makefile now includes modules/bsd_zfs/bsd_zfs_sources.mk
unconditionally (defines the shared solaris/zfs lists) and, for
conf_zfs=openzfs, additionally includes modules/open_zfs/open_zfs_sources.mk.
The provider modules register the "zfs" capability during placeholder import,
exactly like the java placeholder, so no include collision occurs.

Build-structure refactor only; the same objects build for each mode. Kernel
impl-agnostic bits (openzfs_cv_timedwait, the -DCONF_ZFS_OPENZFS redirect, the
OpenZFS patch-apply) stay in the kernel/top-level Makefile.

Validated under KVM:
  conf_zfs=bsd     -> EXIT=0, cpiod finished, pool populated, libsolaris.so.

… pool at root

In open_zfs mode the ZFS pool was not usable as the root filesystem: the
guest mounted an empty pool at / and powered off with "Failed to load
object: /hello" (issue reported by wkozaczuk on PR cloudius-systems#1423).  bsd_zfs
worked end to end.

Root cause: the per-object flag that makes mkfs pick the OpenZFS
pool-root mountpoint convention,

    $(out)/tools/mkfs/mkfs.o: CXXFLAGS += -DCONF_ZFS_OPENZFS

was placed in the early conf_zfs=openzfs block, ABOVE the line where
`out` is defined (out = build/$(mode).$(arch)).  At that point
$(out) expands to the empty string, so the target-specific variable
bound to the target `/tools/mkfs/mkfs.o` instead of the real
`build/release.x64/tools/mkfs/mkfs.o`, and the define never reached
the compile.  mkfs.cc therefore always took its #else (BSD) branch:

    zpool create -f -R /zfs osv        # no -m /

With BSD ZFS the pool-root mountpoint defaults to / so the pool mounts at
the -R altroot /zfs, and cpiod --prefix /zfs/ populates the osv dataset;
the loader then mounts osv and finds /hello.  OpenZFS instead defaults the
pool-root mountpoint to /<pool> (/osv), so under -R /zfs the pool mounted
at /zfs/osv.  cpiods /zfs/... writes then landed on the builders ramfs
(nobody was mounted at /zfs) and were lost on shutdown, leaving the on-disk
osv dataset empty apart from the osv/zfs child mountpoint stub.  At boot the
mount + pivot both succeeded, but every lookup in the root ZAP returned
ENOENT, so /etc/fstab and /hello were invisible.

Fix: define a plain make variable (conf_zfs_openzfs) in the early block and
attach the -DCONF_ZFS_OPENZFS per-object flag AFTER `out` is defined,
gated on that variable.  mkfs.o now compiles with the define, so open_zfs
uses:

    zpool create -f -R /zfs -m / osv   # mountpoint / -> mounts at /zfs

and cpiod populates the osv dataset exactly as BSD does.  The pool now
mounts at / and the app runs.  Kernel stays impl-agnostic; the only
consumer of CONF_ZFS_OPENZFS is the userspace mkfs build tool, and the
change is gated on conf_zfs=openzfs so conf_zfs=bsd is byte-for-byte
unchanged.

Validated under KVM, both modes booting native-example
end to end from the ZFS root:
  conf_zfs=bsd      -> zfs: mounting osv from device /dev/vblk0.1; /hello runs
  conf_zfs=openzfs  -> ZFS: root mounted ok; /dev,/proc,/sys remount; /hello runs
Write+readback of a file at the ZFS root and its survival across a reboot
confirm the mount is a real, persistent ZFS root.

…fs=openzfs

The six ZFS tests that drive the OpenZFS userspace C API via
dlopen("libzfs.so") -- tst-zfs-recordsize, tst-zfs-direct-io,
tst-zfs-encryption, tst-zfs-db-sim, tst-zfs-trim and misc-zfs-mmap-bench --
plus the libzfs* user-space libraries they need, only exist in
conf_zfs=openzfs.  They were, however, added to the manifests for any fs=zfs
build, which broke conf_zfs=bsd two ways:

 1. usr.manifest emitted /usr/lib/libzfs_core.so (and libzutil/libshare/
    libtpool) gated on $(filter zfs,$(fs_type)).  conf_zfs=bsd does not build
    those four libraries, so the image build failed at upload time with
    "ERROR: file upload failed: ... No such file or directory:
    './libzfs_core.so'".

 2. The dlopen tests were in the always-on `tests` list, so a conf_zfs=bsd
    test.py run collected them from usr.manifest even though libzfs.so /
    libzfs_core.so are not present -- they could only ever print
    "SKIP: cannot load libzfs.so" (noise), and tst-zfs-trim in particular
    looked like it stalled.

Split those six tests into zfs-openzfs-only-tests and append them (and the
zfs-user-libs manifest emission) only when conf_zfs=openzfs, matching how the
top-level Makefile builds the libraries.  tst-zfs-crucible-stress, misc-zfs-io
and misc-zfs-arc do not dlopen libzfs and stay in both modes.

conf_zfs reaches this Makefile through the environment (scripts/build exports
every name=value build argument); default it to bsd here to match the
top-level Makefile when the module Makefile is invoked directly.

The system-lua/luarocks fallback added to modules/lua/Makefile is a build
convenience unrelated to the selectable BSD/OpenZFS ZFS work in this PR.
Restore the file to its upstream/master state so it is not carried by cloudius-systems#1423;
it can be proposed on its own.

…is busy")

Building a ZFS image ends with the zfs_builder running
  /tools/mkfs.so; /tools/cpiod.so --prefix /zfs/;
  /zfs.so set compression=off osv; /zpool.so export osv
and printing "cannot export 'osv': pool is busy" at the very end.

mkfs auto-mounts the pool root (osv at /zfs) and the osv/zfs child
(at /zfs/zfs), and cpiod writes into them.  zpool export's own
zpool_disable_datasets() unmounts datasets it finds in /etc/mnttab, but OSv's
mount shim never populates mnttab, so it finds nothing to unmount; the kernel
then refuses the export with EBUSY because the still-mounted datasets hold the
objsets (spa refcount != 0).  The message is cosmetic -- the image is fully
written -- but it looks like a failure.

Drain the pool explicitly before exporting: unmount the datasets by mountpoint
(children first) with the OSv-native /tools/umount.so, which calls umount(2) ->
VFS_UNMOUNT -> the real zfs_umount -> dmu_objset_disown, dropping the spa
refcount to zero so the export succeeds cleanly.  This is mode-independent and
avoids the bsd "zfs unmount" command, which needs statfs2mnttab (absent from
the builder image).  umount.so is already built into tools; add it to the
zfs_builder bootfs so it is available there.

… ZIL replay (sync=standard durability)

Bump modules/open_zfs/openzfs from zfs-2.4.2 (6330a45b) to zfs-2.4.3
(83020cf8, tag zfs-2.4.3). 2.4.2 -> 2.4.3 is an upstream PATCH release;
all 29 OSv platform-layer patches (0001..0029) apply cleanly against the
2.4.3 source with no reroll required. Verified: the build-time patch-apply
stamps and libsolaris.so + loader.elf link cleanly under conf_zfs=openzfs,
and conf_zfs=bsd is unaffected.

Also update the pinned branch in .gitmodules, the Makefile pin comment, and
the PERF doc version reference from 2.4.2 to 2.4.3.

OSv's taskq_wait() enqueued a single barrier task and drained just that
barrier. On the ZFS system_taskq (8 worker threads) a free worker runs the
barrier while the other workers are still mid dnode_sync/dbuf_sync, so
taskq_wait() returned early and the syncing thread raced the still-running
workers, corrupting dbuf/dirty records (arc_write_done VERIFY3 / lfmutex
owner asserts under bulk write).

Add taskqueue_drain_all() (wait until the queue is empty AND no worker is
active, with a wakeup when it quiesces) and use it in taskq_wait(), matching
illumos semantics (wait until no task is queued or active). This also handles
ZFS's recursive within-task enqueues: the enqueuing task stays active so
tq_active never empties until the whole recursion completes.

…on OSv

Parallel ARC eviction dispatches arc_evict_task to an AS0 taskq; a forked
PostgreSQL backend that triggers eviction races it (fault on eva->eva_ml,
arc.c:4088). Force zfs_arc_evict_threads=1 (inline eviction), matching the
existing OSv serialization patches (0021/0022).

…09 so the series applies clean)

Patch 0030 (force single-threaded inline ARC eviction on OSv) carried three
hunks against module/zfs/arc.c, but two of them were byte-identical duplicates
of content already added by patch 0009 (zfs: fix ARC sizing, memory pressure
and OSv platform ...):

  hunk cloudius-systems#1 @@ -4096  arc_evict_thread_init(): OSv force zfs_arc_evict_threads=1
                    -- 0030's legitimate, unique change. KEPT.
  hunk cloudius-systems#2 @@ -4756  adds uint64_t arc_reduce_target_size_noshrink(...)
                    -- identical to the function 0009 already adds. REMOVED.
  hunk cloudius-systems#3 @@ -7980  small-RAM arc_c_min cap (allmem < 256 MiB -> allmem/16)
                    -- identical to the cap 0009 already adds. REMOVED.

Because 0009 applies before 0030, the noshrink helper and the arc_c_min cap are
already present in module/zfs/arc.c by the time 0030 runs. 0030 then tried to
add them a second time, so "git apply" of the full series rejected on a clean
tree and the OpenZFS-mode build failed. This was masked on already-built trees
because modules/open_zfs applies the series once and records a
.osv-patches-applied stamp, so a re-checkout never re-ran the apply; the
aarch64 clean-build exposed it.

Fix: drop hunks cloudius-systems#2 and cloudius-systems#3 from 0030, leaving only the arc_evict_thread_init
change, and correct 0030's Subject / commit message / diffstat to describe only
that change (it was over-scoped / mislabeled). The noshrink helper and the
arc_c_min cap remain, exactly once, from patch 0009.

Verified: the full series 0001..0030 now applies with zero rejects from a
pristine (pinned) OpenZFS submodule; libsolaris.so builds in both
conf_zfs=openzfs and conf_zfs=bsd modes; the OpenZFS kernel boots; and
arc_reduce_target_size_noshrink + the arc_c_min cap are present exactly once in
the applied tree.

[cloudius-systems#1423/OpenZFS]

Signed-off-by: Greg Burd <greg@burd.me>

…de (ZIL-replay mount crash)

A znode loaded via zfs_zget() outside the OSv VFS vget() path has no attached
vnode (z_vnode == NULL), so ZTOV(zp) == NULL. ZIL replay at import/mount hits
this: zfs_replay_truncate -> zfs_space -> zfs_freesp -> zfs_trunc/
zfs_free_range/zfs_extend each call vnode_pager_setsize(ZTOV(zp), ...), which
on OSv is vp->v_size = size, so a NULL vnode page-faults during mount and
aborts the guest before PostgreSQL starts. Intermittent (only when the ZIL has
an un-synced TX_TRUNCATE to replay); became reachable via patch 0029.

Guard each vnode_pager_setsize() with a NULL-vnode check. A replay znode has no
page cache to size; z_size + SA are already authoritative. Combined fork+ZFS
image now reaches ready on 25/25 consecutive boots (-smp 8 -m 32G KVM rs=8k lz4)
vs aborting at mount ~2/3 of boots before.

Classification: [cloudius-systems#1423/OpenZFS].

Signed-off-by: Greg Burd <greg@burd.me>

…D fallback

The OSv SPL platform layer (patch 0001) only recognized x86_64: isa_defs.h
#errored out on any other ISA, and simd.h's non-x86 feature-detection stub
block was missing zfs_sha512_available(). On x86_64 OpenZFS resolves SHA-512
availability through the sha512-x86_64.S asm path, so the stub was never
exercised; aarch64 has no such asm hook wired into the SPL, so icp/algs/sha2
calls zfs_sha512_available() and fails without the fallback.

Add patch 0031 to teach the SPL layer about aarch64:

  - isa_defs.h: add an __aarch64__ ISA branch (LP64, _SUNOS_VTOC_16) mirroring
    the x86_64 branch; endianness is still derived from __BYTE_ORDER.
  - simd.h: add zfs_sha512_available() to the non-x86 B_FALSE stub block so the
    SHA-512 impl selector falls back to the generic C path on aarch64.

x86_64 is unaffected: both hunks add code guarded by __aarch64__ or the
existing non-x86 #else, leaving x86_64 preprocessor output byte-identical.
The submodule stays pinned to the pristine upstream zfs-2.4.3 tag; this is a
new entry in the modules/open_zfs/patches series, not a submodule re-pin.

BUILD-UNVERIFIED on aarch64 - needs an aarch64 metal build+boot to
validate before un-drafting.

Signed-off-by: Greg Burd <greg@burd.me>

open_zfs_sources.mk unconditionally listed the x86_64 ICP crypto asm objects
(aes_amd64.o, aes_aesni.o, sha256/sha512-x86_64.o) plus aeskey.o. The
aes_amd64.S / aes_aesni.S / aeskey.c sources carry no top-level __x86_64__
guard and emit raw x86 instructions, so on an aarch64 build the assembler
chokes. (The raidz vdev_raidz_math_* objects already coexist across arches
because each is #ifdef-guarded to compile-to-empty on the wrong ISA; the ICP
asm objects are not.)

Gate the object selection on $(arch) (open_zfs_sources.mk is already included
after arch is set, so no Makefile change is needed):

  - x64: aeskey.o + the four x86_64 ICP asm objects, exactly as before.
  - aarch64: the ARMv8 sha256/sha512 asm (both #if defined(__aarch64__)
    guarded) and the native ARMv8 blake3 asm, all present in the upstream
    zfs-2.4.3 submodule; no x86 objects, no aeskey.o.

x86_64 is unaffected: with arch=x64 the emitted openzfs-icp / openzfs-icp-asm
lists are byte-identical to before (verified by expanding both branches).

BUILD-UNVERIFIED on aarch64 - needs an aarch64 metal build+boot to
validate before un-drafting.

Signed-off-by: Greg Burd <greg@burd.me>

libc/arch/aarch64/atomic.h pulls in OSv kernel-only headers (the
FreeBSD-derived machine/atomic.h and the bsd/cddl opensolaris sys/types.h),
which are only on the include path for kernel/bsd objects. Userspace
translation units that resolve <atomic.h> to this file (the OpenZFS
libspl/libzfs sources on aarch64) do not have them and do not need them.
Gate those includes on the kernel build and use the compiler atomic builtin
in the userspace path. The x86_64 variant has no such includes, so this is
aarch64-only and leaves x86_64 byte-identical.

The generic-timer frequency sanity check rejected anything above 1GHz. Real
server ARM cores run higher: some server ARM cores run at 1GHz and up; one Neoverse-V1 part
(Neoverse-V1) passes a 1.05GHz generic-timer frequency to the guest, which
tripped the old ceiling and aborted the boot. Raise the ceiling to 2GHz to
cover current cores with headroom while still rejecting a garbage read.